The Sally Clark BN (Fenton)

The Sally Clark BN (Fenton)

Sensitivity analysis?

Precise probabilism

Run with a single probability measure

Imprecise probabilism

Run with various measures, especially focusing on extreme plausible values

The worries

  • No clear principles for choosing ranges

  • No weighting of options

Probabilistic opinion pooling

Independence preservation failure on linear pooling

\(p(X) = p(Y) = p(X\vert Y) = 1/3\)

\(q(X) = q(Y) = q(X\vert Y) = 2/3\)

\(r =p/2+q/2\)

\(r(X\cap Y) = 5/18 \neq r(X)r(Y)=1/4\)

Linear expert deference

\(\mathsf{P}(A=B) <1\)

\(\mathsf{P}(r\vert A=a) = a \,\,\,\, \mathsf{P}(r\vert B=b) = b\)

\(\forall a,b \, \mathsf{P}(r \vert A =a, B = b) = \alpha a + \beta b\)

The hard truth (Gallow)

For any proposition \(r\), if \(A\) and \(B\) are random variables taking values in the unit interval, there is no probability measure \(\mathsf{P}\) for which these conditions hold

Synergy

The reasonable range assumption

For any group of peers whose credences in a proposition \(X\) range from \(x\) to \(y\), the aggregated credence is within \([x, y]\)

Example of synergy

A doctor is fairly confident that a treatment dosage for a patient is correct (.97) and considers the opinion of a colleague, whose credence is 0.96

Higher-order and honesty

Precise estimate

This type of fiber occurs in 25% of carpets in this town…

Estimate with uncertainty

…based on a random sample of 40/100/200 carpets.

Higher-order and honesty

SC with HOP

SC with HOP

SC with HOP: Likelihood ratios

Weight of evidence

Basic intuitions

  • Items of evidence leading to different expected values should be able to have the same weight

  • Items of evidence leading to the same value should be able to have different weights

  • In simple set up, such as Bernoulli trials, weight should increase with the number of observations

  • For unimodal distributions, the wider the distribution associated with a given piece of evidence, the less weight this evidence has

Modular approach

  • weights as associated with distributions

  • weight of evidence results from distribution weight comparison

Shannon information

  • The right path is: 011.

  • There are \(m=8\) possible destinations by decisions at \(\log_2(8)=3\) forks

  • Initially you thought the probability that it is the right one was .5

  • Now you know it is the right one. Surprise: \(\frac{1}{.5}=2\)

  • One bit of information: \(\log_2\left(\frac{1}{.5}\right)=1\)

  • Complete instruction: \(\log_2\left(\frac{1}{.5^3}\right)=3\)

  • Notice that \(\log_2\left(\frac{1}{a}\right)= - \log_2(a)\), so: \(h(x) = - \log_2 \mathsf{P(x)}\)

Entropy: average Shannon information

\(H(X) = \sum \mathsf{P}(x_i) \log_2 \frac{1}{\mathsf{P}(x_i)} = - \sum \mathsf{P}(x_i) \log_2 \mathsf{P}(x_i)\)

Conceptualization

The expected amount of information you receive once you learn the value of \(X\).

Weight of evidence: explication

Absolute distribution weight (adw)

The more informative a piece of evidence is, as compared to the uniform distribution, the more weight it has, on scale 0 to 1.

\(\mathsf{adw(posterior)} = 1 - \left( \frac{H(\mathsf{posterior})}{H(\mathsf{uniform})}\right)\)

Relative distribution weight (rdw)

\(\mathsf{rdw(posterior, prior)} = 1 - \left( \frac{H(\mathsf{posterior})}{H(\mathsf{prior})}\right)\)

Weight of evidence (wDelta)

\(\mathsf{wDelta}(\mathsf{posterior, prior}) = \vert \mathsf{adw}(\mathsf{posterior}) - \mathsf{adw}(\mathsf{prior})\vert\)

Weights of beta distributions

Weights of beta distributions

Weights of beta distributions

Weights of beta distributions

Weight in SC with HOP

Expected weight and completeness

Back to synergy

Back to synergy

Pooling vs expansion: simulated accuracy

Wrapping up

The higher-order approach

  • Leads to more honesty in uncertainty assessment
  • Is more sensible than sensitivity analysis
  • Integrates with Bayesian data analysis
  • Leads to an information-theoretic account of evidential weight
  • Is computationally feasible